• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    System and method to represent the tactile interaction that uses a set of ultrasound transducers. The method (100) comprises receiving (110) a target field (102); calculating (120) one or more pressure focal point trajectories (514) of varying intensity in such a way as to minimize the difference between a result field (516) produced by one or more pressure focal point trajectories and the target field (102); and controlling (130) a set of ultrasound transducers (620) in space-time modulation mode to simultaneously represent one or more calculated trajectories (514) of pressure focal points. The invention accurately represents the dynamic interactions produced in a virtual environment with a variable force or pressure field. (Machine-translation by Google Translate, not legally binding)
    公开号:ES2849963A1
    申请号:ES202030150
    申请日:2020-02-21
    公开日:2021-08-24
    发明作者:Héctor Barreiro；Stephen Sinclair；Tristán Miguel Angel Otaduy
    申请人:Universidad Rey Juan Carlos；
    IPC主号:
    专利说明:

    [0001] SYSTEM AND METHOD TO REPRESENT TOUCH INTERACTION USING A SET OF ULTRASOUND TRANSDUCERS
    [0003] Field of the invention
    [0005] The present invention is within the scope of systems and methods for representing tactile interactions using arrays of ultrasound transducers.
    [0007] Background of the invention
    [0009] Ultrasound haptics produce direct touch sensations on the skin in the air (that is, without the need to hold or use a haptic device), employing a set of ultrasound transducers as actuators that generate high-frequency pressure waves in space around the device. Sets in phase of ultrasound achieve stimulation in the air through a phenomenon known as acoustic radiation pressure. Multiple ultrasound transducers, producing an ultrasound wave of the same frequency, are phase modulated to achieve the maximum combined pressure intensity at a given location in space, known as the focal point. The focus of the ultrasound wave is determined by its wavelength (eg 8.5 mm for 40 kHz ultrasound).
    [0011] Therefore, by modulating the activation of the transducers, it is possible to aggregate the pressure waves at specific points in space, thereby creating pressure focal points. The pressure reaches perceptible values at such focal points and produces a tactile sensation in the air. Pressure waves produce a complex mechanical interaction in the skin, both in space and time, and this mechanical interaction produces an activation and aggregation of mechanoreceptor signals to form tactile perceptions. The interaction with fluids in a virtual environment is an interesting phenomenon that is represented with ultrasound haptics, since they can be moved without restricting the movement of the user.
    [0013] Iwamoto et al. [1] demonstrated for the first time the ability to use phase ensemble ultrasound to focus sufficient pressure of acoustic radiation to induce a haptic sensation in a localized area of the hand, exploiting this principle for haptic stimulation in the air limited to a fixed focal point. Subsequently, Hoshi et al. [2] extended this achievement to vary the focal location over time, generating moving focal points by temporarily varying the modulation of the transducers, producing the sensation of a moving stimulus. Although the induced deformation of the skin is slight, haptic detection is ensured by modulating the field intensity at a frequency at which the skin mechanoreceptors, mainly Pacinia corpuscles, are sensitive, approximately 150 to 250 Hz. counts the location of the focal point as stable in relation to this frequency range, this method later became known as amplitude modulation (AM).
    [0015] These early works used a previously calculated time-varying solution for the ultrasound phases given a desired spatial amplitude distribution. Because these phases must be determined by a non-linear quadratic optimization procedure, the calculations were not capable of real-time control. Long et al. [3] designed an efficient solution that takes advantage of the linearity of the complex-value eigenproblem associated with the desired focal point intensities.
    [0017] Korres and Eid [4] pointed out that the efficiency of this solution supports a different visualization method, in which the amplitudes of the focal points are relatively stable, but their location is modulated at much higher rates. This was called space-time modulation (STM) by Kappus and Long [5], and they showed that such trajectories could produce recognizable tactile shapes. Frier et al. [6] found that in STM there is an optimal focal point speed (between 5 m / s and 8 m / s) to maximize skin sensitivity. Recently, Frier et al. [7] analyzed the combined influence of spatial and temporal sampling on STM.
    [0019] Therefore, there are currently two main control methods for controlling ultrasound devices: amplitude modulation (AM) and space-time modulation (STM). AM controls the position and intensity of the pressure of the focal points to produce pressure distributions on the skin, while STM controls the trajectories of the focal points to draw shapes on the skin. In both cases, high-level control of focal points is translated into low-level control of transducer activation patterns through well-established optimization methods [2], [3].
    [0020] By eliminating the need to hold or use a haptic device, ultrasound haptics allow you to represent interactions with virtual environments for a more immersive and scalable virtual touch experience. To represent the virtual touch interactions, the previous works simply present contact locations at maximum intensity, either through AM or STM, without considering the force distributions produced in the virtual interactions. In [8], a target pressure field is extracted from the interaction of a virtual hand with a fluid dynamic simulation, and then the AM controls are optimized to induce a better matching pressure field in the user.
    [0022] However, AM suffers from some limitations. First, the intensity of the focal points modulates a pressure wave that induces a perceptible vibration in the skin (usually at 200 Hz). Furthermore, STM covers larger areas of skin with higher perceived intensity, by taking advantage of the constructive interference of focal point trajectories with the skin waves they induce. However, all previous work commanded STMs with constant intensity focal point trajectories that do not represent a better match for dynamic interactions. Neither method above calculates STM trajectories of varying intensity to better match dynamic interactions. Not surprisingly, STM poses a more complex challenge than AM. While AM representation can be considered as a quasi-static problem, STM requires the solution to a spatial and temporal problem.
    [0024] Therefore, there is a need for a method to represent tactile interaction using a set of ultrasound transducers that accurately represents the dynamic interactions produced in an environment (virtual or real) with a variable force or pressure field (e.g. , interactions with a fluid that produces a dynamic pressure field in a virtual environment).
    [0026] References
    [0028] [1] T. Iwamoto, M. Tatezono and H. Shinoda, "Non-contact method for producing tactile sensation using airborne ultrasound", in Proceedings of the 6th International Conference on Haptics: Perception, Devices and Scenarios, ser. EuroHaptics 2008. Springer-Verlag, 2008, pp. 504-513.
    [0030] [2] T. Hoshi, M. Takahashi, T. Iwamoto and H. Shinoda, "Noncontact tactile display based on radiation pressure of airborne ultrasound", IEEE Transactions on Haptics, vol. 3, No. 3, pp. 155-165, 2010.
    [0032] [3] B. Long, S. A. Seah, T. Carter and S. Subramanian, "Rendering volumetric haptic shapes in mid-air using ultrasound", ACM Transactions on Graphics (TOG), vol.
    [0033] 33, No. 6, p. 181,2014.
    [0035] [4] G. Korres and M. Eid, "Haptogram: Ultrasonic point-cloud tactile stimulation", IEEE Access, vol. 4, pp. 7758-7769, 2016.
    [0037] [5] B. Kappus and B. Long, "Spatiotemporal modulation for mid-air haptic feedback from an ultrasonic phased array", The Journal of the Acoustical Society of America, vol.
    [0038] 143, No. 3, pp. 1836-1836, 2018.
    [0040] [6] W. Frier, D. Ablart, J. Chilles, B. Long, M. Giordano, M. Obrist and S. Subramanian, "Using spatiotemporal modulation to draw tactile patterns in mid-air", in Haptics: Science, Technology, and Applications, D. Prattichizzo, H. Shinoda, HZ Tan, E. Ruffaldi and A. Frisoli, Eds. Springer International Publishing, 2018, pp. 270-281.
    [0042] [7] W. Frier, D. Pittera, D. Ablart, M. Obrist, and S. Subramanian, "Sampling strategy for ultrasonic mid-air haptics," at the CHI Conference on Human Factors in Computing Systems Proceedings. ACM, 2019.
    [0044] [8] H. Barreiro, S. Sinclair and M. A. Otaduy, "Ultrasound rendering of tactile interaction with fluids", in 2019 IEEE World Haptics Conference (WHC), 2019, pp.
    [0045] 521-526.
    [0047] [9] G. A. Croes, "A method for solving traveling-salesman problems", Operations Research, vol. 6, No. 6, pp. 791-812, 1958
    [0049] Description of the invention
    [0051] The ultrasound transducer assemblies are capable of producing hand-touch sensations, promising hands-free haptic interaction for virtual environments. However, controlling such a set with respect to the reproduction of a desired perceived interaction remains a difficult problem. The present invention performs a dynamic mapping of virtual interactions with existing control methods of ultrasound devices, namely the modulation of the positions and intensities of the focal points over time, a method known as space-time modulation ( STM).
    [0052] In particular, the invention presents an optimization approach that takes into account the known perceptual parameters and the limitations of the STM method. This results in a set of focal point trajectories optimized to better reconstruct a dynamic target pressure field (or a dynamic target force field). The present invention performs path routing optimization for STM, the first method that controls STM ultrasound representing the force distribution resulting from a dynamic virtual interaction. A key aspect of the method is to pose the STM representation as a quasi-static problem, using careful approximations to eliminate the temporal variable at each update of the dynamic representation. As a result, given a target pressure field, the STM representation arises as the computation of focal point trajectories that produce the best-fitting quasi-static pressure field. Then, given a target pressure field, an optimization algorithm calculates the trajectories of the focal points. In one embodiment, the optimization algorithm first initializes the trajectories over the target domain to optimize the pressure intensity-weighted coverage. Then, on a finer scale, the optimization algorithm refines the trajectories to maximize similarity to the target pressure.
    [0054] The present invention relates to a computer-implemented method for representing tactile interaction using an array of ultrasound transducers. The method comprises receiving a target field, wherein the target field is a force field or a pressure field; calculating one or more pressure focal point trajectories of varying intensity such that the difference between a result field produced by one or more pressure focal point trajectories and the target field is minimized; and controlling a set of ultrasound transducers in space-time modulation mode to simultaneously represent one or more calculated trajectories of pressure focal points. Therefore, the method determines one or more pressure focal point trajectories so that the result field produced by these trajectories better matches the target field.
    [0056] In one embodiment, the step of calculating one or more pressure focal point trajectories comprises obtaining one or more initial pressure focal point trajectories and optimizing the one or more initial pressure focal point trajectories by minimizing a set of cost terms based on in the target field and the result field produced by one or more focal point trajectories of Pressure.
    [0058] The initial trajectories are preferably calculated by applying a cluster analysis at the target points to obtain a set of sample points; find a path with a minimum length that passes through all the points in the set; if the path is too long, it is divided into subsets; repeat the last two stages until all paths satisfy a maximum length constraint.
    [0060] Optimization is done by iterating, starting with one or more initial trajectories, pressure focal point trajectories with the objective of minimizing the set of cost terms.
    [0062] The set of cost terms is preferably minimized by iterating down the gradient steps. The set of cost terms can include any of the following cost terms: an intensity cost term to maximize field strength, a length cost term to meet a target path length, an intersection cost term to prevent path samples from approaching a drop distance, a bending cost term to obtain low curvature paths, or a combination thereof.
    [0064] In one embodiment, the result field is obtained by applying a spatial smoothing function to the intensity of the focal points. The spatial smoothing function can be defined as:
    [0066] P ( x) = ^ Pi 0 (| x - x¡M)
    [0068] where p (x) is the intensity of the result field at position x, xi is the closest position to x on a path, pi is the intensity of the target field at position xi, 0 is a Gaussian function and y is a gain heuristics.
    [0070] The method may comprise projecting the target field onto a plane to obtain a plurality of target points defined by a 2D position and a target value. In this case, the method may further comprise undoing the projection to obtain one or more 3D trajectories of pressure focal points.
    [0072] In one embodiment, the method further comprises refining the target field by calculating the intersection of the received target field with a surface of a body. or a part of a user's body. The method may also comprise following the position of the body or body part of the user.
    [0074] In one embodiment, the target field can be generated from a user interaction with a virtual environment (eg, a fluid dynamic simulation). The method may further comprise computing an interactive simulation and extracting the target field from the simulation. In another embodiment, the target field is generated from an interaction of a user with a real environment that is located remotely from the user's location, such as teleoperation (for example, remote control of a sensorized robotic hand) or telepresence. .
    [0076] In one embodiment, each pressure focal point path is a 3D curve defining a closed path. The pressure focal points preferably traverse one or more trajectories calculated at a reference speed between 5 m / s and 8 m / s. Each pressure focal point path preferably satisfies a maximum path length constraint. The number of calculated trajectories of the pressure focal points can be limited to a maximum number of trajectories (eg four trajectories).
    [0078] The present invention also relates to a system for representing tactile interaction using a set of ultrasound transducers. The system comprises a data processing device configured to perform the steps of the method.
    [0080] Another aspect of the present invention relates to a computer program product and / or a computer-readable storage medium for representing tactile interaction using an ultrasound transducer assembly, comprising a computer-usable program code for, when runs on a processor, which performs the steps of the method.
    [0082] Brief description of the drawings
    [0084] A series of figures that help to better understand the invention and that are expressly related to an embodiment of said invention, presented as a non-limiting example thereof, are described very briefly below.
    [0086] Figure 1 represents a flow chart of a method to represent tactile interaction using an ultrasound transducer assembly according to a embodiment of the present invention.
    [0088] Figure 2 shows the steps of pressure focal point calculation trajectories according to one embodiment.
    [0090] Figure 3 shows the steps of pressure focal point calculation trajectories according to another embodiment.
    [0092] Figure 4 is a flow chart for the step of obtaining initial pressure focal point trajectories in Figure 2, according to one embodiment.
    [0094] Figure 5A shows the target pressure points of a target field produced in the user's hand and extracted from a fluid simulation. Figures 5B depict the down-sampled pressure points (clusters) of the target field. Figure 5C shows the optimal initial trajectory for the full set of pressure sample points. Figure 5D shows the optimal paths after dividing the set of sample points into subsets to satisfy a maximum path length constraint. Figure 5E represents the refinement of the trajectories to maximize the intensity of the pressure. Figure 5F shows the resulting reconstructed pressure field obtained by the optimization algorithm.
    [0096] Figure 6 depicts a system for representing tactile interaction using an ultrasound transducer assembly according to an exemplary embodiment.
    [0098] Figure 7 represents the virtual representation of the hand of a user that interacts with virtual columns of smoke used in a dynamic simulation of fluids within a virtual environment.
    [0100] Figure 8A represents the target pressure field produced by the smoke plumes on the surface of the virtual hand. Figure 8B illustrates the reconstructed pressure field produced by the STM representation of dynamically optimized focal point trajectories in accordance with the present method.
    [0102] Figures 9 and 10 represent experimental virtual scenarios (with one or four smoke columns, respectively) and representative examples of reconstructed and target fields using the STM-based method of the present invention and a prior art AM-based method.
    [0103] Description of a preferred embodiment of the invention
    [0105] When spatio-temporal modulation (STM) is used in driving the transducers of a set of ultrasound transducers, a focal point traverses a path in space. The path P can be formally described as a time-dependent position: P = x ( t) e M3. . The perceived intensity of a focal point trajectory is maximized under constructive interference between the movement of the focal point and the propagation of skin waves (Frier et al. [6]). This happens for focal point velocities between 5 m / s and 8 m / s; therefore, a reference speed v is selected within said range for the rendering algorithm. In one embodiment, the selected reference speed v is 7 m / s. Furthermore, to ensure constructive interference in the entire path, the paths are preferably designed as closed paths (ie path P is a closed 3D curve).
    [0107] Frier et al. [6] assume that the path length is given; therefore, the frequency at which the path repeats cannot be controlled independently and depends on the transverse speed and the length of the path. According to preliminary experiments, this has been found to be acceptable up to a maximum path length. Beyond that length, the frequency with which the trajectory repeats is too low, and the stimulus is no longer perceived as continuous. To determine the minimum acceptable frequency (that is, the maximum acceptable path length), circles of different radii were made at the reference transverse velocity of 7 m / s, finding that a minimum frequency of 50 Hz (that is, a length maximum trajectory L = 140 mm) is a safe limit to ensure that the stimulus is perceived as continuous.
    [0109] Typically, haptic devices (i.e., ultrasound transducer assemblies) allow STMs of multiple focal points simultaneously. Each focal point can traverse a different trajectory, with all focal points moving at the reference speed v, and all trajectories satisfy the maximum length restriction L. From preliminary experiments, the number of simultaneous focal points can be limited to a certain number of simultaneous paths (in one embodiment, to four paths). More focal points can achieve greater coverage, but at the cost of a noticeable degradation in perceived intensity.
    [0110] In previous work, STM is used to represent 3D curves, in which the intensity of the radiation pressure of the moving focal point is kept constant along these curves. In the present invention, a temporally and spatially varying pressure field is depicted, in which the intensity of the radiation pressure along the path is locally adapted to the intensity of the pressure field. A trajectory is characterized with a pressure intensity dependent on the position p ( x).
    [0112] Since a focal point cycles multiple times through the same position xi, the represented pressure intensity p (xi) = pi is the same in all cycles. If the trajectory is repeated frequently enough (for example, above 50 Hz), the represented radiation pressure produces a persistent tactile perception. This is equivalent to applying a time-invariant pressure at each position along the path, with its effective magnitude a fraction of the pressure represented. With this assumption, during a window of time, the effective pressure can be considered as a spatially variable but temporally invariant field, that is, a quasi-static pressure field.
    [0114] The effective pressure of the STM plot is not a simple time average of the plotted pressure. According to one embodiment, a perceptual heuristic is followed to approximate this magnitude: represent the same stimuli using amplitude modulation (AM) and STM, and ask subjects to tune the gain and STM until the maximum perceptual intensity is similar. . In practice, a gain y = 1.4 is used.
    [0116] The focal points exhibit a smooth drop determined by the wavelength of the ultrasound signal (eg 8.6 mm for the 40 kHz of a test ultrasound device). As shown by Hoshi et al. [2], this drop can be approximated either by a Gaussian function 0. Based on this finding, together with the heuristic gain y , the effective quasi-static pressure field p ( x) produced by a focal point trajectory can be approximated as :
    [0118] 1 1 ll * - * ill2
    [0119] p OO = - P í 0 (I | z - * í II) = - Pi e 2 ff2; (1)
    [0121] where xi is the closest position to x on the path. The standard deviation or Gaussian decay is set to the same value as the wavelength of the ultrasound signal (eg 8.6mm for 40KHz).
    [0122] The quasi-static pressure field assumption simplifies the rendering algorithm of the present invention. Given a target pressure field, obtained for example from a fluid simulation, the trajectories of focal points are searched whose quasi-static pressure field best reconstructs the target field. To ensure that the quasi-static pressure field assumption is valid, this search preferably meets two constraints, namely that each focal point moves at the reference speed v and the length of each path is equal to or shorter than the maximum length of the trajectory L.
    [0124] Given a target pressure field or a target force field, a path routing optimization algorithm searches for the focal point paths that produce a quasi-static pressure field of best match. FIG. 1 depicts, according to one embodiment of the present invention, a flow chart of a tactile interaction rendering method 100 using an ultrasound transducer assembly implementing a path routing optimization algorithm.
    [0126] The method 100 comprises receiving 110 a target field 102. The target field, which is either a force field or a pressure field, can be calculated externally to the optimization method (eg, by external simulation). The method also comprises calculating 120 one or more pressure focal point trajectories of varying intensity, so that the difference between a result field produced by one or more pressure focal point trajectories and the target field is minimized. Finally, the method comprises controlling 130 a set of ultrasound transducers in space-time modulation mode to simultaneously represent one or more calculated trajectories of pressure focal points.
    [0128] In one embodiment, shown in FIG. 2 , the step of calculating 120 one or more pressure focal point trajectories comprises the following steps: obtaining 202 one or more initial pressure focal point trajectories; and optimizing 204 the one or more initial pressure focal point trajectories by minimizing a set of cost terms based on the target field and the result field produced by the one or more pressure focal point trajectories. Alternatively or in addition to the steps of the embodiment of Figure 2, the step of calculating 120 one or more trajectories of pressure focal points may comprise the steps represented in the flow diagram of Figure 3 obtaining 302, of the field target, a set of sample points with an associated target value and determining 308 one or more initial pressure focal point trajectories using these sample points. In one embodiment, the set of sample points is obtained by applying a cluster analysis on the target field 102 (in particular, a cluster analysis on a plurality of target points of the target field). The cluster analysis can be a weighted k-means algorithm, in which the weights considered are the target value of the target points. The one or more initial pressure focal point trajectories are determined by calculating 304 the shortest closed path that visits all points in the set of sample points to obtain an initial path and, if the length of the initial path exceeds a maximum length of path L, recursively dividing 306 the set of sample points into subsets and calculating the shortest closed path for each subset until the path length of all subsets satisfies a maximum path length constraint.
    [0130] If the embodiments of Figures 2 and 3 are used in combination, the step of obtaining 202 one or more initial pressure focal point trajectories of Figure 2 is implemented according to the steps shown in Figure 3. Alternatively, as illustrated in In the flowchart of Figure 4 , the step in Figure 2 of obtaining 202 one or more initial pressure focal point trajectories may comprise obtaining 402, from the target field, a plurality of target points with an associated target position and value ; and determining 404 the one or more initial pressure focal point trajectories by selecting one or more target point sequences. In this way, the paths can be initialized according to different initialization methods, such as randomly (for example, figure 4), or by applying the grouping and calculation of the shortest path (figure 3). A good initialization makes the optimization algorithm more efficient and / or converges to a better solution. Paths are computed by optimization, and there are optimization methods that require initialization and then refinement, and optimization methods that sample paths and select the best path.
    [0132] According to one embodiment, the optimization problem can be solved in two stages, at two different resolutions, looking for trajectories that maximize coverage and the integrated pressure intensity subject to the trajectory length restriction. The first stage is a rough stage that performs a search global (path initialization), while the second stage is a good stage that performs a local search (path refinement). In another embodiment, only one of these two steps may be necessary. For example, one embodiment may include using only the first stage if the precision provided is sufficient, or using only the second stage if the computing power allows the fine stage to be executed directly.
    [0134] An interactive simulation can be calculated to extract a target pressure field (or a target force field). For example, [8] an interactive fluid simulation can be used, following the user's hand and modeling the hand as a moving obstacle in a 3D simulation of a gaseous medium. As described in [8], fluid dynamics can be modeled using discretized incompressible Euler equations on a 3D Eulerian grid, with semi-Lagrangian advection and massively parallel Jacobi relaxation for pressure resolution. Fluid simulation can be run on a GPU for maximum performance.
    [0136] The method may comprise refining the target field by calculating the intersection of the received target field with a surface of a body or a part of a body of a user. To that end, the method may comprise following the position of the body or the body part of the user. For example, to define the target pressure field, the hand can be voxelized and the voxel positions are selected so that they are visible from the side of the domain that corresponds to the location of the ultrasound device. To simplify the path optimization problem, a plane can be fit to the voxel positions so that the voxel positions are projected onto the plane, making path optimization a 2D problem. Formally, the target pressure field can be described by a set of pressure target points T defined by the 2D positions xi and their corresponding target pressure values p * (xi):
    [0138] T = {(xi E l 2, p '(xi))}.
    [0140] The first rough stage includes the initialization of the path. To ensure high computational performance, the initialization of the trajectories preferably uses only a representative subset of the pressure target points (although the full set of pressure target points T, which require more computational resources, can also be considered). Figures 5A to 5D represent the initialization step according to the embodiment of Figure 3. Starting with a set (or subset) of pressure target points 502 of the target field 102 (eg, pressure target points on the user's hand drawn from a fluid simulation , as shown in Figure 5A), a set 504 of pressure sample points 506 with an associated target value 302 (Figure 5B) is obtained by applying a weighted k-means grouping to the set of pressure target points 502. The Cluster analysis produces a downsampled target pressure field formed by the 506 pressure sample points.
    [0142] Given the set 504 of pressure sample points 506, a set of closed paths visiting all points is sought, subject to the maximum path length L. The optimal solution to this problem may require an arbitrarily large number of paths; however, as noted above, it is advisable to limit the number of paths to a certain number of paths (for example, four paths). Consequently, the resulting trajectories may not visit all pressure sample points, and an optimal set must be selected.
    [0144] This problem is solved iteratively. An optimal path (shortest closed path 508, Figure 5C) that visits all pressure sample points 504 is calculated first 304. If the shortest closed path 508 is too long, the set 504 of pressure sample points 506 is divided into two subsets and optimal paths (shorter closed paths 508 that visit all points in subset 510) are calculated separately. The subsets 510 of the pressure sample points 506 are recursively divided 306 until all the shortest closed paths of all subsets 510 satisfy the maximum length constraint L. If the number of resulting paths is greater than four, the four trajectories with the highest integrated pressure (as in the example in Figure 5D), thus obtaining the initial trajectories 512.
    [0146] The operations for calculating an optimal path for a set of points and for dividing a set of points into additional subsets are described below according to one embodiment, in which the set of points is either a set 504 or a subset 510 of sample points. pressure 506.
    [0148] Given a set of points, find the shortest closed path 508 visiting all the points corresponds to the problem of the traveling salesman. This problem can be solved using the 2-opt algorithm [9], which admits closed trajectories. The 2-opt computing cost sets an upper limit on the size of the set 504 of pressure sample points 506. In one embodiment, a maximum of 50 points is set. Therefore, the weighted k-means grouping stage is executed with 50 or fewer groupings.
    [0150] To divide a set of points, the direction of maximum extent is found, the points along this direction are bounded, and a division plane orthogonal to the direction is placed at the midpoint of the two limits. To find the maximum propagation direction, the covariance matrix of the points is calculated, weighted by their pressure value. The direction of maximum propagation corresponds to the eigenvector with the highest eigenvalue.
    [0152] The second fine stage includes the refinement of the trajectory. After initialization, the paths pass through lobbies and meet the maximum length constraint. However, due to their rough sampling, the initial trajectories are not optimally aligned with pressure peaks and ridges. Path refinement is performed at a higher resolution, sampling each path at N points. In one embodiment, N is set to 20, which sets points 7 mm apart from each other, that is, the distance traveled by a focal point in 1 ms.
    [0154] During refinement, the goal is to maximize the pressure intensity by sampling each path and moving the samples locally to locations with higher pressure, while ensuring that the paths retain the following properties: (i) they satisfy the maximum length constraint; (ii) to achieve maximum coverage, they do not intersect (automatically); and (iii) they do not bend at acute angles, since algorithm design decisions come from perceptual observations on smooth paths, and paths with sharp corners achieve less coverage. To implement refinement, the objective and properties are formulated as cost terms of an objective function, and a minimization algorithm is executed.
    [0156] P = {x¿} = arg min cp. . cb
    [0158] Given the trajectories with samples {x¿ e M2}, we formulate a cost term pressure intensity as:
    [0160] cp = - Z í P * O í ) (2)
    [0162] The pressure intensity cost term (Cp) is minimized as the path samples xi (that is, points that form or define the path) are moved to locations with higher pressure.
    [0164] With a target path length L, and N samples per path, the target length is obtained if the length of each path segment is L / N. Then a length cost term is formulated as:
    [0166] .1 = Z í O l x ^ - X í l l - ^) 2; (3)
    [0168] where xi and xi + 1 are two consecutive path samples.
    [0170] If two trajectories or two portions of a trajectory are closer than the fall distance of the focal points, a, they stimulate the overlapping of areas of the skin. The result can be considered inefficient, since the area covered by the skin is larger if the trajectories are farther apart. An (automatic) intersection cost term that prevents path samples from getting too close is formulated as:
    [0171] 2
    [0172] Ci = Z í jm ax (: - | x; -X i |, 0); (4)
    [0174] where xi and xj are two consecutive path samples.
    [0176] Finally, to favor low curvature trajectories, a bending cost term is formulated as:
    [0178] cb 2 Oi + i-Xj) x (-¡ - j = Si arctan - );
    [0179] (-í + i-- í) T (-¡ - ¡- i); (5)
    [0181] where xM, xi and xi +1 are three consecutive path samples.
    [0183] In one embodiment, the trajectories are optimized by iterating downgrade steps of the four cost terms (although fewer cost terms or even additional cost terms may also be considered). For the pressure intensity term, a 2D grid is established with the pressure target values p * (x) and bicubic interpolation is used to evaluate the pressure at a subgrid resolution and compute robust gradients. In addition, a line search is applied to ensure that the stage along the gradient reduces the cost. Figure 5E shows an example of the trajectories 514 calculated after refinement.
    [0185] To take into account the effective magnitude of the pressure field, a heuristic gain is preferably incorporated and , as explained above in equation (1). The represented pressure of a point on a trajectory is stated as pi = y p * (xi), based on the target pressure field p *. Figure 5F shows the reconstructed pressure field (result field 516) according to the quasi-static pressure model previously analyzed in equation (1).
    [0187] Once the 2D trajectories 514 are fully calculated, they are uploaded to 3D for STM representation on the ultrasound transducer assembly. This is accomplished by undoing the projection of the voxels of the hand.
    [0189] Each path is 140 mm long and traversed in 20 ms at 7 m / s. The STM rendering API of a set of ultrasound transducers updates a burst of X consecutive focal point positions every 1 ms (for example, X is 40 in the Ultrahaptics STRATOS Explore USX). Therefore, each path is sampled linearly at 800 points spaced 0.175 mm apart. These points are fed in groups of 40 points to the STM rendering API of the ultrasound transducer assembly.
    [0191] According to a further aspect of the present invention, there is provided a system for representing tactile interaction using a set of ultrasound transducers. The system comprises a data processing device configured to receive a target field (either a force field or a pressure field); calculating one or more pressure focal point trajectories of varying intensity such that the difference between a result field produced by one or more pressure focal point trajectories and the target field is minimized; and controlling a set of ultrasound transducers in space-time modulation mode to simultaneously represent one or more calculated pressure focal point trajectories (for example, calculating space-time modulation controls corresponding to the 3D focal point trajectories and sending the controls to a set of ultrasound transducers). The data processing device can be configured to implement the method as shown. defined above in any of the different embodiments.
    [0193] Figure 6 depicts a system 600 according to an exemplary embodiment. System 600 comprises a data processing device 610, such as a computer with a processor and a GPU. The data processing device 610 manages the interaction of a user's hand 602 with gaseous fluid media in a virtual environment using a set of ultrasound transducers 620. A tracking device 630 is configured to track the position of the body or a part of the user's body. In the example shown, the tracking device 630 is a manual tracking device, such as a Leap Motion controller, that monitors the position of the user's hand 602.
    [0195] Both the tracking device 630 and the ultrasound transducer assembly 620 are connected (eg, via USB) to the computer running a simulation (eg, a fluid dynamic simulation). The ultrasound transducer assembly 620 is driven by a computer controlled driver circuit 640. The system may comprise actuator circuit 640 configured to operate ultrasound transducer assembly 620. The system may also comprise ultrasound transducer assembly 620. The system may comprise tracking device 630 to track the position of the body or part. of the user's body (in this particular case, to follow the position of the user's hand 602).
    [0197] Data processing device 610 implements the method as defined in any of the preceding figures. The calculated phase delays and amplitude values in STM mode are sent from the data processing device 610 to the actuator circuit 640. The data processing device 610 can be configured to calculate an interactive simulation (eg, fluid dynamic simulation ) and extract the target field 102 from the simulation. In this case, the target field 102 is generated from an interaction of a user's hand 602 with a virtual environment. The computer receives the position of the hand and updates the target field 102 generated in the user's hand 602 based on the position of the hand in the virtual environment.
    [0199] The interactions produced during the virtual simulation can be displayed on a display device 650, such as a monitor, configured to represent the interactive simulation in a virtual environment 704. Figure 7 represents the virtual representation (ie, virtual hand 702) of user hand 602 interacting with virtual plumes of smoke 706 used in fluid dynamic simulation. Smoke plumes 706 generate a pressure target field 102 in virtual hand 702 in virtual environment 704.
    [0201] Data processing device 610 calculates target field 102 and pressure focal point trajectories 514 of varying intensity that produce a pressure field of best match in the user's hand, in accordance with the method described. Figure 8A depicts the target pressure field 102 produced on the surface of the hand by the smoke plumes of Figure 7, which sets the target for the method optimization algorithm. Figure 8B illustrates the reconstructed pressure field, result field 516, produced by the STM representation of the dynamically optimized focal point trajectories.
    [0203] In such an interaction, haptic perception is dictated by a spatially and temporally varying pressure field on the skin, which is used as a target for the optimization algorithm. Figures 9 and 10 compare the reconstruction quality of the present representation method against the representation method of [8]. Figures 9A and 10A depict a 3D representation of a fluid dynamic simulation with one smoke plume 706 and four smoke plumes 706, respectively; Figures 9B and 10B depict respective target field 102 generated in virtual hand 702; Figures 9C and 10C show the respective result field 516 represented by the present method, while Figures 9D and 10D illustrate the reconstructed field 902 represented by the AM-based method of [8]. These results show that the method of the present invention manages to provide a larger and more uniform coverage than the AM-based method of [8]. In particular, AM produces ambiguous results when representing the interaction with a wide column (Figure 9D) or with multiple thin columns (Figure 10D), whereas the current STM-based method suffers from no such ambiguity.
    权利要求:
    Claims (34)
    [1]
    1.- A method to represent the tactile interaction that uses a set of ultrasound transducers, characterized in that the method (100) comprises:
    receiving (110) a target field (102), wherein the target field (102) is a force field or a pressure field;
    calculate (120) one or more pressure focal point trajectories (514) of varying intensity such that the difference between a result field (516) produced by one or more pressure focal point trajectories and the target field (102 ) is minimized; and
    controlling (130) a set of ultrasound transducers (620) in space-time modulation mode to simultaneously represent one or more calculated trajectories (514) of pressure focal points.
    [2]
    2. The method according to claim 1, characterized in that the calculation step (120) of one or more trajectories (514) of pressure focal points comprises:
    obtaining (202) one or more initial pressure focal point trajectories; and
    optimize (204) the one or more initial pressure focal point trajectories by minimizing a set of cost terms based on the target field (102) and the result field (516) produced by the one or more pressure focal point trajectories .
    [3]
    3. The method according to claim 2, characterized in that the set of cost terms is minimized by iterating steps of gradient descent.
    [4]
    4. The method according to any of claims 2 to 3, characterized in that the set of cost terms includes an intensity cost term (c p ) to maximize the field intensity.
    [5]
    5. The method according to any of claims 2 to 4, characterized in that the set of cost terms includes a length cost term (ci) to meet a target path length (L).
    [6]
    6.- The method according to any of claims 2 to 5, characterized in that the set of cost terms includes an intersection cost term (ci) to prevent the trajectory samples from approaching a fall distance (a) .
    [7]
    7. The method according to any of claims 2 to 6, characterized in that the set of cost terms includes a bending cost term (cb) to obtain low curvature trajectories.
    [8]
    8. The method according to any of claims 2 to 7, characterized in that the step of obtaining (202) one or more initial trajectories of pressure focal points comprises:
    obtaining (402), from the target field, a plurality of target points (502) with an associated position and a target value; and
    determining (404) the one or more initial pressure focal point trajectories by selecting one or more target point sequences (502).
    [9]
    9. The method according to any of claims 1 to 7, characterized in that the calculation step (120) of one or more trajectories of pressure focal points comprises:
    obtaining (302), from the target field, a set (504) of sample points (506) with an associated target value; and
    determine (308) one or more initial trajectories (512) of pressure focal points by:
    calculating (304) the shortest closed path (508) that visits all the points of the set (504) of sample points (506); and
    recursively divide (306) the set (504) of sample points (506) into subsets (510) and compute the shortest closed path (508) for each subset (510) until the path length satisfies a length constraint maximum trajectory for all subsets (510).
    [10]
    10. The method according to claim 9, characterized in that the set of sample points is obtained by applying a cluster analysis on a plurality of target points (502) of the target field (102).
    [11]
    11. The method according to claim 10, characterized in that the cluster analysis is a weighted k-means algorithm.
    [12]
    12. The method according to any preceding claim, characterized in that the result field is obtained by applying a spatial smoothing function to the intensity of the focal points.
    [13]
    13. The method according to claim 12, characterized in that the spatial smoothing function is defined as:

    [14]
    The method according to any preceding claim, further comprising projecting the target field onto a plane to obtain a plurality of target points defined by a 2D position (x,) and a target value.
    [15]
    The method according to claim 14, further comprising undoing the projection to obtain the one or more pressure focal point trajectories.
    [16]
    16. The method according to any preceding claim, further comprising refining the target field by calculating the intersection of the received target field with a surface of a body or a part of a body of a user.
    [17]
    17. The method according to claim 16, further comprising monitoring the position of the body or part of the body of the user.
    [18]
    18.
    [19]
    19.
    [20]
    twenty.
    [21]
    twenty-one.
    [22]
    22. The method according to any preceding claim, characterized in that each pressure focal point trajectory satisfies a maximum trajectory length restriction.
    [23]
    23. The method according to any preceding claim, characterized in that the number of calculated trajectories of pressure focal points is limited to a maximum number of trajectories.
    [24]
    24. The method according to any preceding claim, further comprising calculating an interactive simulation and extracting the target field of the simulation.
    [25]
    25.- A system to represent tactile interaction using a set of ultrasound transducers, characterized in that the system (600) comprises a data processing device (610) configured to:
    receiving (110) a target field (102), wherein the target field (102) is a force field or a pressure field;
    calculate (120) one or more pressure focal point trajectories (514) of varying intensity such that the difference between a result field (516) produced by one or more pressure focal point trajectories and the target field (102 ) is minimized; and
    controlling (130) a set of ultrasound transducers (620) in space-time modulation mode to simultaneously represent one or more calculated trajectories (514) of pressure focal points.
    [26]
    26. The system according to claim 25, further comprising an actuator circuit (640) for actuating the set of ultrasound transducers (620).
    [27]
    27. The system according to claim 26, further comprising the set of ultrasound transducers (620).
    [28]
    28. The system according to any of claims 25 to 27, further comprising a tracking device (630) to track the position of the body or part of the body of the user.
    [29]
    29. The system according to claim 28, characterized in that the tracking device (630) is a manual tracking device.
    [30]
    30. The system according to any of claims 25 to 29, characterized in that the target field (102) is generated from an interaction of a user with a virtual environment.
    [31]
    31. The system according to any of claims 25 to 30, characterized in that the data processing device (610) is configured to calculate an interactive simulation and extract the target field (102) from the simulation.
    [32]
    32. The system according to claim 31, further comprising a display device (650) configured to represent the interactive simulation in a virtual environment (704).
    [33]
    33.- A computer program product to represent tactile interaction using a set of ultrasound transducers, characterized by comprising a program code usable by computer to carry out, when executed on a processor, the steps of the method defined in any of the Claims 1 to 24.
    [34]
    34. A computer-readable storage medium comprising instructions for causing a processor to carry out the steps of the method defined in any of claims 1 to 24.
    类似技术:
    公开号 | 公开日 | 专利标题
    JP6452257B2|2019-01-16|Method and apparatus for generating a sound field
    Barfoot et al.2004|Motion planning for formations of mobile robots
    US20190197840A1|2019-06-27|Grouping and Optimization of Phased Ultrasonic Transducers for Multi-Field Solutions
    KR20200017418A|2020-02-18|Robot system and method for generating reaction force to implement virtual boundary
    Schwartz et al.2000|Arm trajectory and representation of movement processing in motor cortical activity
    Cheung et al.2008|Animal navigation: general properties of directed walks
    US10026015B2|2018-07-17|Imaging control to facilitate tracking objects and/or perform real-time intervention
    US20210037332A1|2021-02-04|Algorithm Enhancements for Haptic-Based Phased-Array Solutions
    Kimmel et al.1998|Multivalued distance maps for motion planning on surfaces with moving obstacles
    US11263878B2|2022-03-01|Phase computing device, phase computing method, haptic presentation system, and program
    ES2849963B2|2022-01-05|SYSTEM AND METHOD TO PRESENT TACTILE INTERACTION USING A SET OF ULTRASOUND TRANSDUCERS
    Chen et al.2018|Redirected walking strategies in irregularly shaped and dynamic physical environments
    Ogunmolu et al.2016|Vision-based control of a soft robot for maskless head and neck cancer radiotherapy
    Sminchisescu et al.2002|Building roadmaps of local minima of visual models
    Barreiro et al.2019|Ultrasound rendering of tactile interaction with fluids
    Alvarez et al.2014|3d robot formations planning with fast marching square
    Riahi et al.2020|Noncontact steering of magnetic objects by optimal linear feedback control of permanent magnet manipulators
    Park et al.2004|Haptic rendering with predictive representation of local geometry
    US11036230B1|2021-06-15|Method for developing navigation plan in a robotic floor-cleaning device
    CN111338384B|2021-06-08|Self-adaptive path tracking method of snake-like robot
    Page et al.2006|Ridge-valley path planning for 3D terrains
    Chiroiu et al.2018|On the trajectories control of a hybrid robotic system
    Erwin et al.1993|A comparison of models of visual cortical map formation
    Li et al.2020|Mapping various large virtual spaces to small real spaces: A novel redirected walking method for immersive VR navigation
    Lassak2020|Fast Decision-making under Time and Resource Constraints
    同族专利:
    公开号 | 公开日
    WO2021165555A1|2021-08-26|
    ES2849963B2|2022-01-05|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    CN105426024A|2015-11-25|2016-03-23|吉林大学|Ultrasonic focus based haptic feedback system and method|
    WO2018200424A1|2017-04-24|2018-11-01|Ultrahaptics Ip Ltd|Algorithm enhancements for haptic-based phased-array systems|
    WO2019031057A1|2017-08-07|2019-02-14|ソニー株式会社|Phase computation device, phase computation method, tactile sensation presentation system, and program|
    法律状态:
    2021-08-24| BA2A| Patent application published|Ref document number: 2849963 Country of ref document: ES Kind code of ref document: A1 Effective date: 20210824 |
    2022-01-05| FG2A| Definitive protection|Ref document number: 2849963 Country of ref document: ES Kind code of ref document: B2 Effective date: 20220105 |
    优先权:
    申请号 | 申请日 | 专利标题
    ES202030150A|ES2849963B2|2020-02-21|2020-02-21|SYSTEM AND METHOD TO PRESENT TACTILE INTERACTION USING A SET OF ULTRASOUND TRANSDUCERS|ES202030150A| ES2849963B2|2020-02-21|2020-02-21|SYSTEM AND METHOD TO PRESENT TACTILE INTERACTION USING A SET OF ULTRASOUND TRANSDUCERS|
    PCT/ES2021/070075| WO2021165555A1|2020-02-21|2021-02-01|System and method for representing the tactile interaction employed by an array of ultrasound transducers|
    [返回顶部]